System and method for determining anisomelia condition of a subject using image analysis and deep neural network learning

ABSTRACT

The present invention is an Deep Neural Network based technology relating to diagnosis of Anisomelia, also referred to as Leg Length Discrepancy (LLD). This invention is a system and method, which comprises of a diagnosis device referred to as the “LEG-Minder” device that is typically installed in a diagnosis center setting, and diagnoses for LLD on the basis of a neural network model with patient&#39;s leg photos or x-rays thereof; and a neural network learning server referred to as the “LEGislator” which is connected to the Internet and performs Deep Neural Network (DNN) learning, on the individual LLD databases generated by a plurality of the “LEG-Minder” device(s). In particular, the present invention relates to a technology in which patient&#39;s leg photos (or x-rays) and the corresponding diagnostic result data are acquired in each diagnosis center and then individually uploaded to the LEGislator; and then, on the basis of the uploaded information the LEGislator performs DNN learning to generate an upgraded neural network model, which is then disseminated to the “LEG-Minder” device(s), providing them the latest learnings, which subsequently helps in improving the diagnosis accuracy. This invention optimizes the diagnosis environment of the diagnosis center for Anisomelia.

FIELD OF INVENTION

Anisomelia or Leg length discrepancy (LLD) is a widespread and commoncondition involving abnormal loading of the lower extremity and lumbarjoints. While simple X-rays can resolve between 70% and 80% ofdiagnostic problems, only one-third of the world's population hasdiagnostic imaging access. This proposed solution elegantly strengthensthe processes for the assessment, adoption, and use of appropriatehealth technologies for diagnostic imaging with simple digitalphotographs (e.g., from a smartphone) and avoids the expensive need forX-ray equipment or can become a substitute for cases that need radiationprotection of the public, workers, patients, and the environment.

Anatomic leg-length inequality is near-universal—Leg length discrepancy(LLD) affects up to 90% of the general population with a meandiscrepancy of 5.2 mm. In most such cases, LLD is mild (<20 mm). Whenoverlooked during early medical examinations, spinal severe cordmisalignment and kyphosis occur in children born with this condition.So, early detection is vital in Anisomelia. This system addresses aidingin early-detection for every human being. While simple X-rays canresolve about 75% of diagnostic problems, nearly half of the world'spopulation has no diagnostic imaging access per World HealthOrganization). Inexpensive non-radiology equipment to perform a simpletest to detect LLD is very much needed. Hence the need for the systemthat can detect this. Most valuable features for new inspection methodlike the proposed system here will be:

a. No radiation exposure

b. Equal or more accurate than current inspection methods

c. Is cheaper

The present invention is a technology relating to a Leg LengthDiscrepancy diagnosis system using Deep Neural Network (DNN) learning,which comprises of a diagnosis “Leg-Minder” device that is installed ineach diagnosis center and determines the patients legs to either have aheight discrepancy or not on the basis of a neural network model with apatient leg photograph or a potential radiographic image (eg X-rayimage) as inputs; and a neural network learning server “LEGislator” thatis connected to the internet and performs DNN Learning on the LLDdatabase of a plurality of “LEG-Minder” devices in the network.

In particular, the present invention relates to a technology in whichpatient's leg photos and diagnostic result data are acquired in eachdiagnosis center and then uploaded to the neural network learning server“LEGislator device”. Then on the basis of this information the learningserver performs DNN learning on its neural network model so as togenerate an upgraded neural network model. This upgraded model is laterdownloaded to all the LEG-Minder devices in the network. By the aboveconstitution, the LEG-Minder device becomes a part of a neural networkmodel, which is optimized to the diagnosis environment within adiagnosis center.

BACKGROUND ART

Anisomelia is classified as either anatomical (structural) orfunctional. Structural is side-to-side differences in lower limb length,while functional is due to bio-mechanical abnormalities of jointfunction in the lower limbs (athletes). The causes for LLD can beCongenital or can be acquired. Congenital causes include phocomelia anddysgenetic syndromes. Acquired causes include: dysplasias, Ollier'sdisease, polio & osteomyelitis, neurofibromatosis; septic arthritis;fractures; and also surgically induced. LLD can exist from childhood orit can develop in adult life. The clinical methods (direct and indirectmethods) in common use to measure leg length discrepancy (LLD) cannotalways meet the demands of precision and accuracy based on numerousstudies. Some of the current clinical methods of assessing thisdiscrepancy include tape measures, planks and blocks to level thepelvis, and x-rays including scanograms and ultrasounds. Studies haveshown clinical assessments of the examiners were incorrect by greaterthan 5 mm in 29% of subjects.

In addition, it also turns out that these methods are:

-   -   Expensive (both time-wise and $-wise)    -   Not necessarily prescribed to every patient due to exposure to        radiation of the pelvic region.

The diagnosis alone can be complicated. In simple words, it involves:

1) The attending physician needs to clearly observe the patients postureand suspect LLD condition

2) The physician should have the presence of mind to initiate theradiological process for the patient

3) The clinician reading the x-rays has to accurately classify this tobe a potential LLD problem and

4) The patient has to follow through the long process and stick with itto complete the process.

LLD diagnosis and detection is not a part of regular annual medicalcheck-ups for anyone and especially so for younger children in the agegroup 5-12. When LLD is not identified and fixed early, posturedeformation, gait asymmetry, and lower-joint damages can occur in lateryears. Apparent LLD condition is more common than true LLD, some of thesymptoms for which include:

a. Scoliosis

b. Flatfeet

c. Unleveled hips

In cases where the apparent LLD cannot be confirmed via X-rays, thisproposed system and method is an indispensable method for accuratediagnosis.

Some biases are prevalent and endemic in medicine. Such biases couldresult in deeply fallible and flawed medical diagnoses/data. This flaweddata and decisions can amplify harm caused to the complex human bodysystem. Since the initial screen is done under the experience and skillby a practitioner, the accuracy of the first examination, whichsometimes may be affected by screeners' personal condition, is thetrigger for the course of action a patient/physician takes. Therefore,its important for this to be accurate and to ‘de-bias’ via qualitativeand quantitative means.

As discussed above, the analysis and classification of LLD can becomplex and sometimes overlooked even for a trained eye. This gap isaddressed by using Machine Learning (ML). ML can be used to performcomputationally complex tasks leading to determination of certainconditions in high-risk patients. This invention uses ML technologies todiagnose LLD. The computational means provides consistent and reliablefirst diagnosis results without relying solely on the skills of thescreeners and the associated problems as discussed in the aboveparagraph. This makes the invention an elegant and easier solution todiagnose LLD with improved/speedier diagnosis creating an economicallong-term solution to diagnose the LLD condition.

Further, in the current art, individual diagnosis centers areintroducing various technologies in their own tests yet, which rendersdiagnosis technology inconsistent and insufficiently reliable. Becauseeach of large diagnosis centers individually utilizes diagnostic resultdata of their own, process can be quite complex. An electronic means toprovide uniformity to the solution removes complexities arising fromvarious biases. Often Artificial Intelligence (AI) is used for theseprocesses as increasingly complex diagnosis can be automated to not missthe intricate details.

DISCLOSURE OF INVENTION Technical Problem

The present invention is proposed with reference to the above-mentionedproblems in the Background Art. The objective of the present inventionis to provide an elegant, speedy, and accurate LLD diagnosis system andmethod, using computational approaches such as Deep Neural Network (DNN)learning, in which diagnosis accuracy of devices can be be graduallyimproved. This is performed by a client-server type architecture with anInternet-connection without individually modifying each of neuralnetwork models. This invention uses Deep Neural Learning Network bywhich computers may think and learn like a human constantly andcategorize objects and hence is Artificially Intelligent.

Technical Solution/Algorithm

Image classification techniques are used by computer vision tasks (e.g.,segmentation, object detection, and image classification) and patternrecognition exploiting handcrafted features from a large-scale database,thus allowing new predictions from existing data.

In the ML algorithm associated with this invention, images are parsedinto multiple layers and computationally higher-level features areextracted from the raw input images. Progressively, algorithm is trainedon pre-classified images and is the validated on a separate set ofpre-classified images. From the predictions on the training images, MLalgorithm compares the expected result and charts an auto correctionsequence. The ML algorithm thus learns from existing data and derives amodel which is then used to predict new images presented to it.

Mathematically, the ML algorithm uses Convolutional Neural Network (CNN)transforms, which apply functions such as convolution, kernelinitialization, pooling, activation, padding, batch normalization, andstride to the images for processing. The CNN then adaptively learnsvarious image features, and performs an image transformation, focusingjust on the features that are highly predictive for a specific learningobjective. Leveraging such patterns, classifiers like sigmoid andSoftmax classifiers are then applied to learn the extracted andimportant features. This results in a Neural network model that can beused to make predictions on test or patient leg images.

The current method involves using pipelines for LLD classification usingML techniques to develop a lightweight CNN model for automatic detectionof LLD in various bilateral leg pictures or bilateral leg x-rays. Thislightweight model is then adopted into the LEG-Minder devices.

The ML models were trained using several simulated LLD image datasetwith different parameters and filters in the LEGislator. With everyiteration, hyperparameters are fine-tuned, activation functions areoptimized to improve the accuracy of the model. Then, binaryclassification is employed for detection. This model is deployed in theLEG-Minder devices. Periodically, the LEG-Minder database is uploaded tothe LEGislator which then performs an upgrade on its neural networkmodel version. Then the upgraded model is deployed again into theLEG-Minder devices. From a LEG-Minder device standpoint, the end usercan simply upload a normal photograph of the legs or a radiograph or aphotograph in a bilateral fashion and feed it to the ML algorithm, whichthen compares and classifies the image for LLD detection. This issummarized in the proposed user-flow as shown in FIG. 11.

Advantages

The framework proposed here is practical and can be easily compared tohandcrafted measurements by practitioners. The potential outcomes ofthis invention can be applied to expand into other areas of ML basedclassifications in the medical field as well as specifically alsomeasure the discrepancy more accurately. The work can also be extendedto predict results on plain photographs (non-xray). For example, in agroup of legs—as in a class picture type of setting in a school) toquickly identify potential issues and alert the parents to seek furthermedical help and aid in early detection. The proposed solution iselegant enough to be applied in the context of reaching young childrenin underprivileged and underserved communities by identifying LLD earlyenough with a simple photograph which can be uploaded remotely into theLEG-Minder device with appropriate controls via a local medicalpractitioner as opposed to getting more expensive X-rays and qualifiedtechnicians to read them. This also helps prevent expensive deformitieslater in the life with a trigger to seek medical help for the necessaryintervention early enough. The current system and method here offers aelegant, speedy, and accurate LLD diagnosis system using computationalapproaches:

-   -   When compared to the existing clinical methods (direct and        indirect methods) this computational method will be more        precise, can take advantage of the continual learning to update        the knowledge and is computationally more accurate than human        estimation and measurements. Inaccurate diagnosis by a human        doctor can lead to higher illness burden on patient and well as        the hospital, and insurance companies. This innovation helps        reduce the patient and hospital burden.    -   The fact that this algorithm can even work with non-radiological        images and can use normal photographs will enable the use of        technology even for those patients that cannot tolerate exposure        to radiation of the pelvic region.    -   The use of normal photographs also allows the invention to be        used in under-served and under privileged communities where        radiography reach isn't available. This invention will spread        the reach of the LLD diagnosis to nearly half the world's        population that does not have yet access to radiological        equipment.    -   The global radiology gap is far less discussed than        infectious-disease outbreaks and natural disasters, but its        dangers to public health are every bit as urgent!! In certain        countries, there is a serious deprivation of radiologists. The        use of normal photographs will allow the population to use this        technology that can provide more accurate diagnosis based on        normal photographs and technicians can be trained to operate it        vs more qualified radiologists.    -   Time taken from the point of first patient-doctor contact to the        time diagnosis is completed can be significantly reduced with        this invention. Since there is no scheduling to be made with        radiology department, patient having to come back at another        time for getting X-rays, once taken, waiting for the Radiologist        to read the X-rays and then passing the information to the        orthopedic doctor can all be cut down to a few minutes as the        technician or the orthopedic doctor themselves can use the        device to diagnose LLD problem in a matter of few minutes.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 Overall system and Claims illustrates the overall systemincluding LEG-Minder and LEGislator for Leg Length discrepancydiagnosis.

FIG. 2 System Level Architecture is a block diagram of the LEG-Minderand the LEGislator combination according to the present invention.

FIG. 3 Component Level architecture is a block diagram of the LEG-Minderdevice which forms the patient image portion of the invention.

FIG. 4 LEGislator Master Neural Network Server is a block diagram of theLearning server which forms the cloud-based algorithm updater portion ofthe invention.

FIG. 5 Step A Initial Learning depicts the steps involved with initiallearning by LEGislator

FIG. 6 Step B Upgrade LEG Minder depicts the process of upgrading theLEG_Minder by the LEGislator

FIG. 7 Step C: Transfer Data depicts the steps involved with LEG-Mindertransfer of data

FIG. 8 Step D Transfer Learning depicts the steps involved with transferlearning by LEGislator

FIG. 9 Step EMission Mode depicts the steps involved with operationalmode of LE-Minder

FIG. 10 Pipeline Architecture for Neural Network Model Generation is ablock diagram of the Learning server's pipeline architecture.

FIG. 11 Proposed user flow with the LEG-Minder device is a flow chart ofhow the user would use the LEG-Minder device.

FIG. 12 Convolution and pooling shows the configurations and the summaryof transformations.

FIG. 13 Convolutional layers and Max Pooling shows the traversal of animage though all the convolutional and the pooling layers.

FIG. 14 Image Pre-Processing Flowchart shows the first step in thealgorithm which is pre-processing of input.

FIG. 15 Convolution Building shows the architecture of the convolutionalneural network.

FIG. 16 Padding Implementation shows how padding is implemented in thealgorithm.

FIG. 17 Dimensionality Reduction shows how performance improvements areimplemented to achieve dimensionality reduction.

FIG. 18 Deep Neural Network shows the implementation of back propagation

ALGORITHM SUMMARY

-   -   1. As a first step, the algorithm pre-classifies generated        images into a training set and a validation set.    -   2. Next, the algorithm addresses image normalization. Since the        input raw data may not already be normalized, i.e., there is no        control over what pixel size a user may input, the algorithm        rescales the images to normalize all the input parameters. After        this step all the images will have identical parameters although        the content inside may be very different.    -   3. Then the algorithm pre-processes the image data. Image        pre-processing is done by augmentation of the existing data and        considers overfitting the data.    -   4. Then, the algorithm expands the scope of the input images to        account for yet unseen image variations by amending the existing        images to overfit while training the data, by use of various        transforms like rotation, flipping, skewing, relative zoom, and        other affine transformations such as translation, rotation,        isotropic scaling and shear.    -   5. Next, the algorithm will set a specific batch size to process        a batch of images at once.    -   6. Then, the algorithm will invoke a binary class mode because        there are only two classes for classification—namely the LLD or        non-existing LLD case.    -   7. Then the algorithm will pass the training image set via        convolutions to learn particular features of the training        images.    -   8. The algorithm then will initiate pooling and the image        traversal through the above path of convolutions while        extracting the next set of features.    -   9. The algorithm will then stack multiple sets of convolutions        and pooling layers as described in the prior two steps. The size        of the subject image is progressively reduced, which is then fed        into the dense layers.    -   10. The algorithm also implements a softmax classifier to reduce        binary loss through the above process steps.    -   11. Learning rate is then adjusted for convergence to arrive at        a solution.    -   12. The algorithm will then output a single neuron with a        sigmoid activation which gives the final result on the processed        image.    -   13. Next, the algorithm will setup the same on a validation        dataset to verify that the algorithm is classifying accurately.    -   14. The algorithm is now trained and can accept a new image upon        which it will process the feature extraction identified earlier        to finally arrive at a classification output on the single        neuron.        Algorithm Details:        Image Pre-Processing (Step 3 in Algorithm):

Image preprocessing refers to step 3 defined in the algorithm summary.The system can accept various image formats, such as JPEG, GIF, PNG,etc., typically used for photographic images. Formats such as DICOM,NIFTI, and Analyze AVW are used in medical imaging. Formats such asTIFF, ICS, IMS, etc., are used in microscope imaging. Image data will bestored as a mathematical matrix.

Approximately, 2D image of size 1024-by-1024 is stored in a matrix ofthe same size. It takes an image as an input and recognizes image pixelsand converts it into a mathematical matrix. As shown in FIG. 14, thealgorithm then checks for the image's compatibility with predictivemodel. If the image is compatible, it is fed directly to the imagesegmentation section skipping the rescaling. During the rescalingoperation, first, a combination of linear filters and non-linear filtersare used to remove the undesirable properties in the input image. Imageenhancement, if needed, is accomplished either in spatial or frequencydomain as necessary. In the Image Segmentation section, the image issegmented to separate the background and foreground objects. All theobjects are marked with different markers setting a clear path for thepredictions using the ML model. The transformed image is then stored inthe database and fed to a predictive model for further processing. TheImage Preprocessing flowchart shows a detailed outline of the process.

Convolutional Neural Network Layers (Steps 7, 8, 9) in Algorithm:

Refer to FIG. 15, and steps 7, 8, 9 in the algorithm summary. In theconvolutional neural network, the neurons in the first convolutionallayer are not connected to every single pixel in the input image, butonly to pixels in their receptive fields. In turn, each neuron in thesecond convolutional layer is connected only to neurons located within asmall rectangle in the first layer. This architecture is selected sothat it allows the network to concentrate on small low-level features inthe first hidden layer, then assemble them into larger higher-levelfeatures in the next hidden layer, etc.

A neuron located in row i, column j of a given layer is connected to theoutputs of the neurons in the previous layer located in rows i toi+f_(h)−1, columns j to j+f_(w)−1, where f_(h) and f_(w) are the heightand width of the receptive field as shown in FIG. 16 paddingimplementation. Zero padding is implemented such that a layer has thesame height and width as the previous layer by adding zeros around theinputs.

In cases where the input image layer is to be connected to much smallerlayer, technique to space out the receptive fields is implemented sothat the model's computational complexity is dramatically reduced. Thisis new and special about the algorithm because it is not a genericmethod to implement stride. Since there is no guarantee what the inputimage would look like, an illustration for a 5×7 input layer (with zeropadding) to connect to a 3×4 layer, using 3×3 receptive fields and astride of 2 is illustrated in FIG. 17 Dimensionality Reduction. Here,the stride is the same in both directions, but it may not necessarily beso with the input images. A neuron located in row i, column j in theupper layer is connected to the outputs of the neurons in the previouslayer located in rows i×s_(h) to i×s_(h)+f_(h)−1, columns j×s_(w) toj×s_(w)+f_(w)−1, where s_(h) and s_(w) are the vertical and horizontalstrides.

Filters in Convolution Neural Network Layers (Steps 7, 8, 9):

A neuron's weights, which are referred to as filters or convolutionkernels are assigned as a small image which is equal to the size of thereceptive file. The first filter is a vertical filter which is a squarematrix full of 0s except for the central i^(th) column, of 1s); Thecorresponding neurons will ignore everything in their receptive fieldexcept for the central vertical line. This technique will ensure thatthe horizontal white lines get enhanced while the rest gets blurred. Thesecond filter is a horizontal filter, which is again a square matrixfull of 0s except for the central j^(th) row, of 1s); The correspondingneurons using these weights will ignore everything in their receptivefield except for the central horizontal line. This technique will ensurethat the vertical white lines get enhanced while the rest gets blurred.During training the convolutional layer the algorithm will automaticallylearn the useful filters for its task of processing an image, and thelayers described above will learn to combine them into more complexpatterns. This allows this algorithm to stack such simple filters. Suchcombination of filters will be inputs in each convolution and the outputis one feature map per filter. It has one neuron per pixel in eachfeature map, and all neurons within a given feature map share the sameparameters. Neurons in different feature maps use different parameters.Thus, a convolutional layer simultaneously applies multiple trainablefilters to its inputs, making it capable of detecting multiple featuresanywhere in its inputs. All neurons in a feature map share the sameparameters, thus dramatically reducing the number of parameters in themodel. Once the CNN has learned to recognize a pattern in one locationof the image, it can recognize it in any other location within theimage. Sometimes, in case of normal human leg photographs, images arecomposed of multiple sublayers: one per color channel. This case isillustrated in FIG. 18 Regular photograph with three colors. At a basiclevel, there are red, green, and blue (RGB) while grayscale images havejust one channel. When some of the latest photography techniques areused, some images may also have extra light frequencies (such asinfrared).

In this case, a neuron located in row i column j of the feature map k ina given convolutional layer l is connected to the outputs of the neuronsin the previous layer l−1, located in rows i×s_(h) to i×s_(h)+f_(h)−1and columns j×s_(w) to j×s_(w)+f_(w)−1, across all feature maps (inlayer l−1).

In order to compute the output of a given neuron in a convolutionallayer, the following formula is used:

$z_{i,j,k} = {b_{k} + {\sum\limits_{u = 0}^{f_{h} - 1}{\sum\limits_{v = 0}^{f_{w} - 1}{\sum\limits_{k^{\prime} = 0}^{f_{n^{\prime}} - 1}{x_{i^{\prime},j^{\prime},k^{\prime}} \times w_{u,v,k^{\prime},k}\mspace{14mu}{with}\mspace{14mu}\left\{ \begin{matrix}{i^{\prime} = {{ixs}_{h} + u}} \\{j^{\prime} = {{jxs}_{w} + v}}\end{matrix} \right.}}}}}$

In this equation:

-   -   1. z_(i,j,k) is the output of the neuron located in row i,        column j in feature map k of the convolutional layer (layer l).    -   2. s_(h) and s_(w) are the vertical and horizontal strides,        f_(h) and f_(w) are the height and width of the receptive field,        and f_(n′) is the number of feature maps in the previous layer        (layer l−1).    -   3. x_(i′,j′,k′) is the output of the neuron located in layer        l−1, row i′, column j′, feature map k′.    -   4. b_(k) tweaks the overall brightness of the feature map k (in        layer l).    -   5. w_(u,v,k′,k) is the connection weight between any neuron in        feature map k of the layer l and its input located at row u,        column v and feature map k′.        Pooling Layers (Steps 7, 8.9 in Algorithm):

Pooling layers shrink the input image in order to reduce thecomputational load, the memory usage, and the number of parameters andthis is specifically done to reduce the possibility of overfitting. Eachneuron in a pooling layer is connected to the outputs of a limitednumber of neurons in the previous layer, located within a smallrectangular receptive field. Its size, the stride, and the padding isdefined, but the pooling neuron will be assigned no weights; all it doesis aggregate the inputs using an aggregation function such as the max ormean. Only the max input value in each receptive field makes it to thenext layer, while the other inputs are dropped. At the end of this step,the image still looks identical to the input image, but the pixeldensity is drastically reduced. This reduces the computations, memoryusage, and the number of parameters. This stage will offer a smallamount of rotational invariance and a slight scale invariance. Thisinvariance is useful in cases where the prediction should not depend onthese details for the classification task.

A few convolutional layers are stacked, and each one is followed by arectified linear activation function or ReLU. This is a piecewise linearfunction to output the input directly if it is positive, otherwise, itwill output zero. This ReLu is used to achieve better performance. TheReLU layer, then a pooling layer, then another few convolutional layersagain followed by ReLU, then another pooling layer is part of thisarchitecture. The input image gets smaller and smaller as it progressesthrough the network, but it also typically gets deeper with more featuremaps. At the top of the stack, a regular feedforward neural network isadded, with a few fully connected layers followed by ReLU and the finallayer outputs the prediction.

FIG. 12 shows the convolutional blocks, with a depth of several layers,it is shown that a set of convolutions were followed by pooling. Theinput image is 300 by 300 pixels. There is a single neuron with asigmoid activation on the output. The summary of the layers is alsoshown with the corresponding size changes. The first convolution reducesthat to 147 by 147. From there, convolution loop repeats until itsreduced to 35 by 35 which is then fed into the dense layers. A total of40,165,409 trainable parameters were identified with this algorithmiteration.

While it is difficult to examine a CNN layer-by-layer, each layer'soutput can be visualized and extracted features seen. This is depictedin FIG. 13.

Softmax Classifier (Step 10 in Algorithm)

Refer to step 10 in the algorithm—the Softmax Regression classifier isused to predict only one class at a time. Even though it is generallyused for multiclass since the outputs are strictly limited to mutuallyexclusive classes, this classification works inherently well for aclear, unequivocal classification. Overall the model is aimed atestimating probabilities and making predictions. The objective for thealgorithm is to overall estimate a high probability ‘p’ for the intendedclass and consequentially a low probability ‘(1−p)’ for the other class.This is accomplished by minimizing the cost function for cross entropy.The cross entropy is designed for measuring how well the estimated classprobabilities matches the target class by penalizing the model when itestimates a low probability for a target class. The Cross entropy costfunction is represented by the mathematical expression:

${J(\theta)} = {{- \frac{1}{m}}{\sum\limits_{i = 1}^{m}{\sum\limits_{k = 1}^{K}{y_{k}^{(i)}\log\;\left( {\overset{\hat{}}{p}}_{k}^{(i)} \right)}}}}$

In this equation:

y_(k) ^((i)) is the target probability that the i^(th) instance belongsto class k. Since the prediction is either Yes or No, it will be a 1 ora 0 depending on whether the instance belongs to the class or not. Ifthe assumptions are wrong, the cross entropy will be greater by anamount called the Kullback-Leibler (KL) divergence. The cross entropy insuch cases, will be governed byH(p,q)=−Σ_(x) p(x)log q(x)

where p and q represent the discrete probability distributions.

The gradient vector of this cost function with regard to θ^((k)) is:

${\nabla_{\theta^{k}}{J(\theta)}} = {\frac{1}{m}{\sum\limits_{1}^{m}\left( {\left( {{\overset{\hat{}}{p}}_{k}^{(i)} - y_{k}^{(i)}} \right)X^{(i)}} \right.}}$

With this, the gradient vector for every class is computed and thenGradient Descent is used to find the parameter matrix Θ that minimizesthe cost function determined by the cost entropy cost function.

Embodiment for Carrying Out the Invention

The invention is described below in detail with reference to theaccompanying drawings.

FIG. 1 Overall system and Claims illustrates the overall systemincluding LEG-Minder and LEGislator for the Leg Length discrepancydiagnosis system using Deep Neural Network (DNN) according to thepresent invention. The DNN section is explained in the section below:

Deep Neural Network Implementation

In order to non-linearly combine information in the server, Dense Neuralnetworks is used. They are used in the server ‘LEGislator’ device.“LEGislator’ can be used on its own to also make categoricalpredictions, although its primarily used to improve the CNN's accuracyin the LEGMinder device (client). So, dense layers are on the server andthey are hierarchically on top of the CNN architecture in the LEGMinderdevices. This allows recombination of the information learned by theconvolutional layers from the clients. This comprises of one passthroughinput layer, one or more hidden layers, and the one final layer calledthe output layer. This is depicted in FIG. 18 Deep Neural Network. Forthis specification, the layers close to the input layer are referred toas the lower layers, and the ones close to the outputs are referred toas the upper layers.

The main components of the algorithm here are:

-   -   Algorithm in the server will handle one mini-batch at a time,        and goes through the full training set multiple times. Each such        pass is referred to in this specification as an Epoch.    -   Each mini-batch is passed to the network's input layer, which        sends it to the first hidden layer.    -   The algorithm then computes the output of all the neurons in        this layer for every Epoch.    -   The result is passed on to the next layer, its output is        computed and passed to the next layer, and so on until it        reaches the output layer. This is the forward pass: it is        exactly like making predictions in the CNN, except all        intermediate results are preserved.    -   Next, the algorithm measures the network's output error    -   Then it computes how much each output connection contributed to        the error using chain rule.    -   The algorithm then measures how much of these error        contributions came from each connection in the layer below all        the way to the input layer. This will be done by propagating the        error gradient backward through the network.    -   Next, Gradient Descent is performed to tweak all the connection        weights in the network, using the error gradients just computed.    -   The Rectified Linear Unit function ReLU(z)=max(0, z) is used for        this algorithm.

Referring to FIG. 1, the individual components comprise of a“LEGislator” based on ML which is connected to the Internet and performsDNN learning on the neural network of the “LEG-Minder” device. Inparticular, the present invention relates to a technology in whichpatient leg photos and diagnostic result data are acquired in eachdiagnosis center by the LEG-Minder device and then uploaded to theneural network learning server “LEGislator”. The learning serverperforms DNN learning and updated the neural network model which in-turnis installed in the ‘LEG-Minder’ of the diagnosis center.

FIG. 2 is a block diagram of the LEG-Minder and the Neural NetworkServer “LEGislator” combination according to the present invention. Thisis a system level architecture representation showing the individualactions/functions that each of the devices would perform.

First, the component level architectures are described for easyunderstanding:

Component Level Description

Leg-Minder Device (100)

FIG. 3 Component Level architecture depicts the LEG-Minder Device (100).This device captures the input data from the patient and is equippedwith neural network model which is preferably implemented as computersoftware or can be customized to a hardware.

The LEG-Minder device (100) captures the patient image either directlythrough a camera or allows uploading of the image (e,g., x-ray image) tothe device using external means into the image processor module (110).In this specification, the neural network model which is initiallyinstalled in the LEG-Minder device (100) is referred to as the “currentneural network model” (120). Further, in this specification, no-LLDrefers to a photo/x-ray of a person that is classified NOT to haveAnisomelia or Leg Length Discrepancy, whereas LLD represents one of anon-ignorable possibility of Leg Length Discrepancy and possiblyrequiring further examination by a specialist Orthopedic fortreatment/rectification. The image is processed to find if the subjectpicture has LLD or Not by using the LLD Diagnosis module (130). When anew image is diagnosed by the LLD diagnosis module (130) thecorresponding computation result, as to whether the subject image hasLLD or not, is stored in the classified Diagnosis Database (140).

Periodically, the LEG-Minder device transfers it local DiagnosisDatabase (140) to the LEGislator though the Neural Network Updatingmodule (150). The Neural Network updating module (150) also receivesupdated model from the LEGislator device (200), and updates the currentneural network model (120). As the neural network updating module (150)updates the current neural network model (120), the version history ismaintained in the “Version Control Module” (170).

There are two databases in the device, namely, the LLD database forstorage of raw images (180), and a diagnosis database (140) consistingof images classified by the model.

Sentry Security module (160) ensure the integrity of the learned modelas well as governs the security aspects related to sentry operationssuch as fending off any malicious attempts to induce bad data either atthe network level or at the image ingress level for the LEG-Minderdevice (100).

Learning Server “LEGislator” (200):

FIG. 4 LEGislator Master Neural Network Server depicts the “LEGislatorDevice” (200). The function of the LEGislator Device is to performtransfer learning on the accumulated dataset, and to generate anupgraded neural network model to be disseminated to the LEG-Minder (100)device(s). The operations of LEGislator Device (200) is orchestrated bythe Learning model orchestrator (210).

Upon initiation by the Learning model orchestrator (210), the LEG-Minderdevice(s) (100), will transfer their diagnosis database (140) to theMaster LLD database (220). The transfer learning processor (240) thenuses Deep Neural Network Model A (230) and the Master LLD database (220)to perform deep learning using DNN techniques to generate a Deep NeuralNetwork Model B (250). Deep Neural Network Model B (250) is theembodiment of all the current learning in this client serverarchitecture invention. The Server version tracker and distributor (280)keeps track of Deep Neural network model A (230) and the Deep Neuralnetwork model B (250). It performs a switch of model B to model A at theappropriate time as well as performs dissemination of the current modelto all the LEG-Minder devices (100) via the Internet, thus performing anauto-upgrade. However, to prevent any spurious devices from gettingupdates or to ensure that no compromised LEG-Minder device (100) evergets the update, the Server security module (260) ensures that properAuthentication, Authorization, Accounting and Auditing is conducted. Forthis purpose, Server security module (260) will work in conjunction withthe Device version tracker (270). The Device version tracker (270) is adatabase that keeps track of every device that connects to the neuralnetwork, its associated credentials and security parameters to ensureintegrity of the overall system.

After updating to the latest neural network model, the LEG-Minderupdates its Version control (170). Depending on certain constraints, theVersion control (170) may choose to accept or reject the downloadedversion from the LEGislature device.

System Architecture:

At a system level the LEGislator and a plurality of the LEG-Minders forma client-server architecture, where in, the LEG-Minder is the client andthe LEGislator is the server. This is depicted in FIG. 2 System LevelArchitecture.

Consider a healthcare network like that of Kaiser Permanente which ispresent in multiple locations and across multiple states within the USA.For example, Santa Clara, Fremont, Irvine, Atlanta etc. Each practicecan cater to a certain number of patients. In the case of an orthopedicdoctor, when the doctor sees a range of patients, they develop a certainlevel of knowledge. Given the regionalities, population densitybelonging to a certain ethic origin etc, they see a certain type ofpatients and they become experts within that population segment and‘know’ what to expect. This based on the specialist's ‘learning’. LA orIrvine might have a different set of patients who bring their ownnuances. So, the “Legminder” is like a regional doctor. The plurality ofthe devices is referring to many such regional doctors who get their ownregional learnings. They get their learning based on the patients theysee.

In the above example, if we replace the regional doctors with one doctorwho serves the entire humanity. Because he or she will see a vast numberof patients, their knowledge base is HUGE. This is a direct relationshipwith the number of patients they see. So, the knowledge has a directcorrelation with the “learning’. In this case, we have a server device,that takes the regional learnings and builds a master database, keepingtrack of the individual learnings (which is like a journal of regionaldoctors). In the case of the Kaiser Permanente scenario, we would rathersee the entire network provide a similar experience to the patients. Forthis to happen in our case, we need every device working based on thesame learning. Therefore server aggregates the learning, develops acommon base from which each client needs to operate and serves thisinformation to the individual client devices.

Since the ‘LegMinder’ is a machine, they need fool proof securitymeasures. Hence the security aspects are embedded to prevent someonefrom providing pictures of donkey's legs versus the expected human legsfor example.

As shown in FIG. 2, the Learning model orchestrator (210) controls theoperations of the LEGislator 200. The Learning model orchestrator (210)performs four fundamental operations. Those are:

-   -   Initial Learning    -   Transfer Learning    -   Database Transfer, and    -   Upgrade

As shown in FIG. 2, the LEG-Minder 100 performs three fundamentaloperations. They are:

-   -   Mission mode (Learn and Predict)    -   Upgrade    -   Transfer database

Each of the operations mentioned above are described in greater detailin—FIG. 5, FIG. 6, FIG. 7, FIG. 8 and FIG. 9 each process step describedin detail as below:

Step A Initial Learning

As shown in FIG. 5, upon receiving the trigger from the learning modelorchestrator (210), the Transfer learning processor (240) will perform:

STEP A1: Initiate learning sequence with its associated image generatorsand create train and validation datasets as described in the AlgorithmSummary.

STEP A2: Generate CNN model by stacking multiple sets of convolutionsand pooling layers along with the dense layers, and a SoftMax classifier

STEP A3: Adjust model fit by adjustable parameters such as the learningrate

STEP A4: Save the resulting Neural network model “B” in the serverversion tracker and distributor (280).

Step B Upgrade LEG Minder

As outlined in FIG. 6 Step B Upgrade LEG Minder, the output of STEP Aabove is used to update the LEG-Minder (100) device. Steps associatedwith this areas described below with reference to FIG. 6:

STEP B0: The LEGislator device (200) will, upon initiation from thelearning model orchestrator (210) initiate the upgrade command toLEG-Minder device(s) (100).

STEP B1: The upgrade process is initiated by the Server version trackerand distributor (280) over an Internet connection.

STEP B2: The LEG-Minder device (100) receives the command into theSentry Security module and authenticates the command received.

STEP B3: Upon authentication, the Neural network updating module (150)will update the current neural network model sent by the LEGislatordevice (200).

STEP B4: Upon successful update verification by Neural network updatingmodule (150), the version control module (170) will update the versionnumber.

STEP B5: The current neural network model (120) is then replaced withthe newly downloaded model.

Step C Transfer Data

At this point the LEG-Minder device has acquired all the new learningsfrom the server and continues to diagnose LLD and update its database asdescribed in the component section. To provide the new learnings back tothe LEGislator server the following steps are used as shown in FIG. 7:

STEP C0: A transfer command is issued by the learning model orchestrator(210) of the LEGislator device (200)

STEP C1: A command is issued to initiate the transfer process by theServer version tracker and distributor (280) over an Internet connectionto specific LEG-Minder device (100).

STEP C2: The Sentry Security module (160) receives it, authenticatesthat it is intended for the correct LEG-Minder device (100) and thenrelays it to the Neural Network Updating module (150).

STEP C3: The Neural Network Updating module (150) initiates upload ofthe diagnosis database (140) to the LEGislator (200) via the Internet

STEP C4: The Server security module (260) ensures that properAuthentication, Authorization, Accounting and Auditing and relayscommand to the Server version tracker and distributor (280).

STEP C5: The Server version tracker and distributor (280) then saves theuploaded images to the Master LLD database (220).

This will initiate the transfer learning process in the LEGislator (200)as shown in FIG. 8.

Step D Transfer Learning

As shown in FIG. 8, STEP D0: A transfer learning command is issued bythe learning model orchestrator (210) of the LEGislator device (200)

STEP D1: The transfer learning processor (240) then uses Deep NeuralNetwork Model A (230) and the Master LLD database (220) to perform deeplearning using DNN techniques.

STEP D2: Above step results in generation of the Deep Neural NetworkModel B (250).

STEP D3: The updated network model is saved in Device version tracker(270) along with the credentials of the LEG-Minder (100) and the Serverversion tracker and distributor (280).

Step E Mission Mode

When the LEG-Minder Device (100) is in Mission mode as shown in FIG. 9,it is ready of diagnosis. When a new image is presented to theLEG-Minder Device (100) for diagnosis of LLD, it performs the followingsteps:

STEP E1: The image processor (110) processes the image per the algorithmdescribed under Algorithm Summary:

STEP E2: Using the current neural network model (120) and the LLDDiagnosis module (130) to process the image, make a prediction and thenupdate the Diagnosis database (140) and the LLD database (180).

Technology Definitions

The “Deep Learning” technology refers to a technology by which computersmay think and learn like a human, especially to group or categorizeobjects and data. Deep-learning methods are representation-learningmethods with multiple levels of representation, obtained by composingsimple but non-linear modules that each transform the representation atone level into a representation at a higher, slightly more abstractlevel. The key aspect of deep learning is that these layers of featuresare not designed by human engineers: they are learned from data using ageneral-purpose learning procedure as used in this invention.

The deep learning is a machine learning technique which is proposed forovercoming the limitation of “Artificial neural network” algorithm. TheDeep learning has two kinds of data categorization approach, i.e.,supervised learning and unsupervised learning. In the supervisedlearning approach, a computer is trained with well-categorizedinformation. This invention uses supervised learning.

Deep learning in this invention uses a pipeline of modules all of whichare trainable. This allows for multiple stages in the process ofrecognizing an object and all of those stages are part of the trainingfor subsequent model generations i.e., representations are hierarchicaland trained.

The invention claimed is:
 1. A computerized method, which isimplementable by using at least: one or more hardware processors thatare configured to execute code, and that are operably associated withone or more memory units that are configured to store code; wherein thecomputerized method comprises: determining whether a particular subjecthas a Leg Length Discrepancy (LLD), by performing: (a1) receiving atraining set of images of legs of patients; (a2) receiving a validationset of images of legs of patients; (b) operating on the training set ofimages by: (b1) performing image normalization and image resizing onsaid images of legs of patients; (b2) modifying the images of thetraining set, by applying one or more image transformation operationsselected from the group consisting of: image rotation, image flip,skewing, zoom modification, isotropic scaling, shear transformation;(b3) performing a binary-type classification of said images of legs ofpatients, into exactly one of: (i) a first class of images that includesonly images that are determined to not be associated with LLD, or (ii) asecond class of images that includes both images that are determined tobe associated with LLD and images that are determined to possibly beassociated with LLD; (b4) passing the images of the training set ofimages via convolutions and extracting a first set of unique featuresfrom said images of the training set; and operating a ConvolutionalNeural Network (CNN) unit which applies convolution, kernelinitialization, pooling, activation, padding, batch normalization, andstride to the images, to detect one or more particular image-featuresthat are determined to be predictive for LLD detection; (b5) performpooling and image traversal, through a particular path of convolutionsthat was passed in step (b4), and concurrently extracting a next set ofunique features from said images of the training set by usingcomputerized-vision object detection and computerized-vision patternrecognition; (b6) stacking multiple sets of convolutions that werepassed in step (b4), and also stacking multiple pooling layers that werepooled in step (b5), to generate reduced-size images; (b7) feeding thereduced-size images into one or more dense layers of said CNN unit; (b8)applying a SoftMax classifier to reduce binary loss, and furtherapplying a sigmoid classifier; (b9) adjusting a learning rate of saidCNN unit for convergence into a solution; (b10) generating by said CNNunit a single-neuron output with a sigmoid activation, which indicates abinary-type output with regard to a particular image; wherein thebinary-type output is either (i) the particular image is not associatedwith LLD, or (ii) the particular image is associated or is possiblyassociated with LLD; (c) operating on the validation set of images by:performing steps (b1) through (b10) on the validation set of images toverify an accuracy of classifications performed by said CNN unit.
 2. Thecomputerized method of claim 1, wherein said images of legs of patientsinclude both (i) X-Ray images of legs of patients and (ii) photographicnon-X-Ray images of legs of patients.
 3. The computerized method ofclaim 1, wherein said images of legs of patients include, exclusively,X-Ray images of legs of patients.
 4. The computerized method of claim 1,wherein said images of legs of patients include, exclusively,photographic non-X-Ray images of legs of patients.
 5. The computerizedmethod of claim 1, further comprising: collecting said images of legs ofpatients at a central server, from a plurality of remote imaging devicesthat are located at a plurality of remote locations; generating aunified Deep Neural Network (DNN) model based on said images of legs ofpatients that were collected from said plurality of remote imagingdevices that are located at said plurality of remote locations; whereinthe DNN model is configured to reduce bias or to eliminate bias indiagnosis of LDD by performing training and convolutions on said imagesof legs of patients that were collected from said plurality of remoteimaging devices that are located at said plurality of remote locations,rather than by relying on legs images from a single source or from asingle hospital or from a single locality.
 6. The computerized method ofclaim 5, further comprising: operating a security module that secures anintegrity of said unified Deep Neural Network (DNN) model from maliciousattacks, and that blocks malicious attacks to introduce bad data (i) atsaid central server at a network level, and (ii) at said plurality ofimaging devices at an image ingress level.
 7. The computerized method ofclaim 6, further comprising: performing a transfer learning process atsaid central server, on a dynamically-updated dataset of images of legsof patients; periodically generating at said central server an upgradedDNN model; and periodically sending the upgraded DNN model to theplurality of imaging devices.
 8. The computerized method of claim 6,wherein said DNN model is configured to detect LLD of a particularperson, based on a group photograph that depicts two or more personsstanding together.
 9. The computerized method of claim 1, wherein saidimages of legs of patients include, exclusively, side images of legs ofpatients, and not frontal images of legs of patients.
 10. Thecomputerized method of claim 1, wherein said images of legs of patientsinclude both: (i) side images of legs of patients, and (ii) frontalimages of legs of patients.
 11. The computerized method of claim 1,wherein the CNN model is developed and is dynamically updated at acentral server computer based on images of legs that are uploaded tosaid central computer server from a plurality of end-user devices;wherein a current version of the CNN model is periodically distributedfrom said central server computer to said end-user devices, anddynamically replaces on said end-user devices a prior version of the CNNmodel; wherein central upgrading of the CNN model, based on images oflegs that are uploaded to said central computer server from a pluralityof end-user devices that are located at a plurality of differentlocations, causes the CNN model and the determining of LLD to be moreresilient to bias.
 12. A computerized system, which is implemented byutilizing at least: one or more processors that are configured toexecute code, and that are operably associated with one or more memoryunits that are configured to execute code; wherein the system comprises:(a) a plurality of distributed end-user devices, wherein each end-userdevice is an electronic device selected from the group consisting of: asmartphone, a tablet, an electronic device comprising a processor and animager; wherein each end-user device is configured to acquire digitalnon-radiological non-X-Ray photographs of legs of persons; wherein eachend-user device is configured to perform: (i) a learn-and-predictprocess, (ii) a Deep Neural Network (DNN) model upgrade process, and(iii) a database transfer process; wherein each end-user devicelocally-stores therein, and locally-runs therein, a local version of aDNN model that is periodically updated by a central computer server; (b)said central computer server, that is configured to communicateseparately, over Internet-based communication links, with each one ofthe plurality of distributed end-user devices; wherein the centralcomputer server comprises a DNN Engine, that is configured to perform:(i) an initial learning process, (ii) a transfer learning process, (iii)a further database transfer process, and (iv) a further DNN modelupgrade process; wherein the DNN Engine periodically upgrades the DNNmodel, and periodically distributes an upgraded DNN model to each one ofsaid end-user devices; wherein at least one of: (I) the plurality of enduser devices, (II) said central computer server, is configured toutilize said upgraded DNN model to generate a determination fordiagnosis, indicating whether or not a particular subject has a LegLength Discrepancy (LLD), by feeding a digital non-radiologicalnon-X-Ray photograph of legs of said particular subject into saidupgraded DNN model, based on output from a sigmoid-activatedsingle-neuron of said DNN model; wherein an accuracy of the diagnosis ofLLD, by each of the plurality of end-user devices, or the said centralcomputer server, gradually improves based on cumulative DNN learning bythe central computer server which is based on analysis of images fromthe plurality of end-user devices.
 13. The computerized system of claim12, wherein the central server computer stores at least: (i) a firstversion of the DNN model, which is currently being utilized for LLDdetermination by at least one end-user device; and also, (ii) a secondversion of the DNN model, which is an upgraded version of the DNN modelthat is more accurate than the first version of the DNN model, and whichis pending for distribution to one or more end-user devices.
 14. Thecomputerized system of claim 12, wherein each end-user deviceperiodically replaces, a current-version of the DNN model that is storedlocally and is utilized locally in the end-user device, with anupgraded-version of the DNN model that is periodically received over anInternet-based communication link from said central computer server. 15.The computerized system of claim 12, wherein each end-user device isequipped with a security module that is configured to block maliciousimages from being added to a locally-stored dataset of images and frombeing copied upstream to said central computer server.
 16. Thecomputerized system of claim 12, wherein the central computer servercomprises: a Master LLD Database which stores images that are utilizedby the central computer server to generate and to update the DNN modelfor detection of LLD; and a Transferred Learning LLD Database whichstores images that were received from a particular end-user device andthat were not yet utilized for updating the DNN model; wherein a DNNModel Updater Unit operates to upgrade or improve the DNN model based onthe images in the Transferred Learning LLD Database; and wherein contentof the Transferred Learning LLD Database is then added to the Master LLDDatabase of the central computer server.
 17. The computerized system ofclaim 12, wherein said DNN model is configured to detect LLD of aparticular person, based on a group photograph that depicts two or morepersons standing together.
 18. The computerized system of claim 12,wherein said images of legs of patients include, exclusively, sideimages of legs of patients, and not frontal images of legs of patients.19. The computerized system of claim 12, wherein said images of legs ofpatients include both: (i) side images of legs of patients, and (ii)frontal images of legs of patients.
 20. The computerized system of claim12, wherein the CNN model is developed and is dynamically updated at thecentral server computer based on images of legs that are uploaded tosaid central computer server from said plurality of end-user devices;wherein a current version of the CNN model is periodically distributedfrom said central server computer to said end-user devices, anddynamically replaces on said end-user devices a prior version of the CNNmodel; wherein central updating of the CNN model, based on images oflegs that are uploaded to said central computer server from a pluralityof end-user devices that are located at a plurality of differentlocations, causes the CNN model and the determining of LLD to be moreresilient to bias.